Nonlinear activations for convolutional neural network acoustic models

Following their triumphs in visual recognition tasks, convolutional neural networks (CNNs) have recently been used to learn the emission probabilities of hidden Markov models in speech recognition. The key distinction of CNNs over deep neural networks (DNNs) is that they leverage translational invariance in the frequency domain, such that weights are shared and there are significantly fewer parameters to train. Since the acoustics of speech indeed display some translational invariance, CNNs could provide more powerful models than DNNs for various speech recognition tasks. Here, we compare the per-frame state classification accuracy of several popular nonlinear activation functions, both sigmoidal and rectified, for a small-scale CNN: the logistic function, the hyperbolic tangent function, the rectified linear unit, the leaky rectified linear unit, and the soft-plus function. We find that the leaky rectified linear unit and soft-plus function perform best by far, which suggests their potential in full-scale CNN acoustic models.